The present invention relates to image display systems, including technology for the creation of aerial real images and other three-dimensional (3-D) effects. More particularly, the present invention involves improved imaging systems, especially useful for computer displays and videoconferencing. More particularly, the present invention involves improved imaging systems, capable of providing multiplane images from video or other sources that simulate the perception of 3-D real depth. The present invention also has the capacity to provide, simultaneously or alternately, multiple images such as for multiple players of a video game.
Numerous attempts have been made over several decades to devise a practical system for 3-D video, as well as 3-D photography in general. The prior art discloses no satisfactory methodology that produces affordable 3-D imaging of an acceptable quality. Additionally no method has been devised for transmission of 3-D images over conventional bandwidths.
Three-Dimensional Imaging Techniques by Takanori Okoshi (1976) analyzes virtually every 3-D imaging method ever devised to that date. Since that time, most 3-D imaging technologies that have been developed use the technologies disclosed in Okoshi's book, and no radically new approaches have been proposed. When discussing the prospects for 3-D television, Okoshi calculated that the bandwidth required to transmit a Lenticular-Sheet 3-D Image, that is the lowest bandwidth required for an autostereoscopic 3-D picture, would be 750 MHz. This corresponds to about 125 of today's conventional TV channels.
Okoshi's analysis shows that neither he nor anyone else knew how the bandwidth requirement could be reduced enough to transmit 3-D video, nor that anyone could imagine what kind of display device could be devised to show it. Okoshi predicted that the next generation of television broadcasting would feature high-resolution, wide-screen display that gave only an “illusion” of depth sensation. To support his prediction, Okoshi cited the beginning of an “epoch” in movies when Cinerama was introduced. Cinerama was a two-dimensional (2-D), curved, wide-screen technique. According to Okoshi, the popularity of Cinerama resulted in a dramatic decrease in efforts to develop other forms of 3-D.
The experience of 3-D is created in the presence of four conditions. The first condition consists of what are collectively called 2-D cues. These cues mainly consist of objects getting smaller, higher, closer together, dimmer, less distinct, less contrasty and less colored as they get further away, as well of course as the fact that foreground objects block the view of background objects. These cues are recorded and reproduced in the course of standard 2-D image recording.
The second condition is parallax. Parallax occurs when a change of position of the viewer produces a different view in which background objects previously hidden by foreground objects become visible. Conventional 3-D stereo techniques lack an ability to convey parallax, but although parallax is not absolutely essential for a viewer to experience 3-D, its presence adds a great deal of realism to 3-D imaging.
The third condition is lateral binocular disparity. This means that the lateral (horizontal) relationship between at least two objects in the scene is different for each eye. This results in different amounts of convergence of the eyes to form a single perceived image of each different object in the scene. It can be reproduced by recording images of a scene from two (or more) different points of view.
The fourth condition is depth disparity, in which at least two objects in the scene are not in focus to the eye at the same time. Thus, accommodation or re-focusing of the eye is required when moved from one such object to the other. This phenomenon, present in real life, is not reproduced with conventional 3-D imaging techniques. It requires focusing of at least two different components of a scene into at least two different depth locations in space.
The inventor has found that this condition is very important because, as the brain acts to refocus the eyes from one depth to the other, it experiences the perception of true depth. Lack of the depth disparity phenomenon leads to eye strain and headaches after extended periods of viewing in current 3-D imaging systems, because of the conflict between accommodation and convergence. Preferably these four conditions correlate to one another when viewing a 3-D image so as to provide the same relationships found when viewing a real life scene.
The popularity of Cinerama, as mentioned above, is especially interesting because it was only a 2-D image projected on a very wide curved screen; yet it produced a realistic depth-containing experience. The effect indicates that something about the display was providing the brain with depth information.
The depth cues present in a Cinerama display were very important and compelling. First of all, a variety of 2-D depth cues were present. As an object gets farther away, it gets smaller, higher up in the frame, less distinct, less color saturated, less contrasty and less bright. Object points get closer together as they recede, such as train tracks appearing to get closer together as they get further away, and foreground objects obscure background objects. Second, the objects depicted on screen were often small compared to the huge screen size. Third, the extremely wide screen necessitated that the viewers focus their eyes differently, the process known as accommodation, when viewing objects at the center of the screen as compared to when they viewed objects on the sides of the screen. This effect was more pronounced the closer one sat to the screen and the smaller each on-screen object was. If an object was nearly as big as the screen, the brain would reduce the viewer's perception of apparent depth. This is because different parts of such an object appeared at different depths, depending on where each part was on the wide screen. However, the brain knew that the entire object should be at only one depth. This caused the brain to lose depth perception with regard to that object and just see it as a curved flat object.
In real life, accommodation is an extremely important, but almost completely ignored, part of what causes depth perception. The eyes constantly refocus on nearer and farther objects in a scene. When focusing on objects at one distance, objects at another distance are seen as being out of focus, and the brain adjusts lens and corneal muscles to bring into sharpest focus whatever one concentrates on. Due to our eyes' limited depth of field, we can never get all objects (or even all parts of any single three-dimensional object) into best focus at any one time. As we shift attention to blurrier objects in an attempt to sharpen their image for clearer recognition, we keep refocusing. Objects keep shifting from clear to blurry, creating a scene in constant flux made of a mix of sharper and blurrier images which keep changing their focus. This effect is even observable with one eye, creating the basis for limited “monocular depth perception.”
While observing a scene, the brain also constantly shifts attention from nearer objects to farther objects in the attempt to merge all perceived double images. When viewing an object at a selected depth, the two eyes are aimed at that object so that the two views of that object overlap precisely, creating a single image in the brain. This is called convergence. At the same time, other objects at other depths do not line up and therefore appear to the brain as double images. As the brain constantly shifts attention among nearer and farther objects, it experiences a continuing flurry of single and doubled images. Through life experience, the brain forms a correlation between each degree of accommodation and each degree of convergence in response to viewing objects at different depths.
In stereoscopy and autostereoscopy, as the object gets farther from the plane of the image, in front or behind, the convergence of the eyes increases, but unlike reality, the accommodation stays the same (since all image information is in one plane). The farther away from the image plane an object is, the larger is the discrepancy between accommodation and convergence. The discrepancy causes the brain to change the convergence and accommodation of the eyes back and forth to create a match between them based on past experience.
In such an experience, fatigue, eyestrain, and headaches result since the objects are, at least in part, not really in focus in the same plane as convergence makes them appear. Also, the further out of the plane of best focus the image appears due to convergence, the harder it is for the viewer to see a 3-D image instead of a double image.
With Cinerama, although depth appeared limited, the appearance of depth was striking because accommodation and convergence seemed to match when looking at different parts of the big screen, because the eyes had to both converge differently and refocus.
This important component of three-dimensional perception (varying accommodation) is not reproduced in prior art 3-D imaging systems. This is because most 3-D imaging is done using stereoscopy or autostereoscopy. In stereoscopy, two images are recorded, one corresponding to the left-eye view of the scene and the other corresponding to the right-eye view. These two images are different, providing what is called “binocular disparity.” This difference forces the viewer's eyes to aim at objects at each selected depth to see them properly. In stereoscopy, each eye is made to view only its corresponding image through the use of an optical device such as red and green glasses, polarized glasses, or lenses which focus one image into each eye, such as in a stereoscope.
Autostereoscopy directs the corresponding images to the eye without the use of any optical device near the person. Instead, optics are located near the images, restricting the angle of view of each image so that each eye still sees only one of the two images. This has been done with lenses, prisms, and light-blocking barriers, for instance. Since the angle of view for each image can be made very narrow, many images can be taken from many angles and viewed one at a time as one moves one's head. This provides an aspect of 3-D perception, called motion parallax, not available from stereoscopy. With motion parallax, one can look around foreground objects to see previously hidden background objects. The images displayed using stereoscopy and autostereoscopy, however, are all in one plane, so the constant refocusing and perception of a mix of sharp and blurry images—which the inventor has found to be so important to the real 3-D experience—is absent. Due to this lack of variable accommodation cues, stereoscopy and autostereoscopy present another difference from reality that is significant.
When the two eyes receive different views of a scene, the brain overlaps them, trying to line the images up exactly. However, two different images viewed from two different angles cannot line up exactly. This binocular disparity gives the brain information about the depth of (or distance to) an object being viewed. If one holds one's thumb in front of some farther object, and focuses on the thumb with both eyes open, one will observe a double image of the farther object. A shift in attention to the farther object causes a double image of the thumb to be observed.
As long as there are some changes in both convergence and accommodation, the viewer's brain perceives a scene as not flat. Once a scene is observed to have depth, a variety of perspective cues complete the illusion of depth, and inform the viewer at what depth each object appears to be.